feat(etl): weekly digest producer — @sigma/report extraction + Monday cron (#167A) - #80

)

* test(ci): gate the fake-D1 doubles, and fail on an unmatched query

midt-bg#325: 24 test files hold 36 `as D1Database` casts, one hand-rolled double each.
Every one dispatches on `sql.includes('…')` and falls through to `{ results: [] }`
when no marker matches, so renaming a CTE or reordering a JOIN leaves the test
green against emptiness — asserting nothing. Only details.test.ts throws today.

This is the acceptance test for that work, written before the work: outside an
explicit allowlist, no file under apps/ or packages/ may type a value as a
D1Database. It is red now (24 files, 36 casts) and goes green when the last
double moves to the shared helper.

The allowlist is by name, never a directory glob — the argument the midt-bg#254 review
already made about the coverage exclusion list. A glob lets a new double leave
the gate by where it sits; a named entry means someone had to add it, which is
reviewable. A stale entry is an error rather than a no-op, so a renamed double
cannot leave the gate widened by a line nobody reads again. That fail-closed
branch is what fires right now, since the helper does not exist yet.

Matching is over blanked source — comments, strings and regex literals removed,
byte positions kept — so a comment describing the old design is not a finding.
`as unknown as D1Database` is matched before `as D1Database` because the short
spelling is a suffix of the long one and a naive pattern counts one cast twice.
`satisfies` is covered too: it is the only other operator that types a literal.

Self-test is mutation-checked — dropping the `as unknown as` alternative, the
`satisfies` alternative, the comment blanking, the trailing word boundary, or
the stale-entry check each kills exactly one named test, and no others.

scripts-test.yml needs no edit: its lane already globs scripts/*.test.mjs.

* test(test-support): a shared D1 double that throws on an unmatched query

The helper midt-bg#325 asks for, as its own private workspace. `@sigma/db` exports only
`.`, and neither apps/etl nor packages/ingest depends on it, so putting the
double under db/src/test/ would have meant a subpath export plus two new
workspace deps. A separate package also sits outside all six measured
workspaces, so it cannot enter their coverage denominators by construction —
stronger than the by-name vitest.shared.ts exclusion the issue proposes, and it
leaves that file (which midt-bg#254 rewrites) untouched.

A route is a marker set and a response; every marker must appear in the SQL, and
the first matching route wins so a specific route can precede a general one.
Unmatched throws, naming the offending statement and every registered marker.
`{ onUnmatched: 'empty' }` buys emptiness back, at the call site, in writing.

Three entry points, one core: fakeD1 for query tests, recordingD1 for the tests
of a *wrapper* over D1 (readonlyD1) that must accept arbitrary SQL and assert on
a call log, throwingD1 for the error paths.

Two design notes worth keeping:

  - `first` is typed `object | null | (call) => object | null`, not `unknown`.
    A top type absorbs the union and the callback form silently loses its
    parameter type — tsc caught it. A D1 row is an object or nothing anyway.
  - No pagination feature. Keyset slicing is already `all: (call) =>
    rows.filter(r => r.id > call.binds.at(-2))`, which is what the doubles in
    companies.test.ts do by hand today.

Tests written before the code, behaviour by behaviour, and mutation-checked:
never throwing on an unmatched all() or first(), matching a route that answers a
different method, `some` for `every` over the markers, last-match instead of
first, no truncation, dropping the marker list from the message, discarding
bind() arguments, not recording prepare() or batch(), and ignoring throwingD1's
supplied error — eleven mutations, each killing a specific named test.

The `?? []` fallback in all() went away rather than getting a test: the route
lookup already guarantees the response is defined, so it was unreachable.
Returning the response instead of the route also keeps `first: null` — a route
meaning "no such row" — distinct from no route at all.

Seventh key in coverage-baseline.json: check-coverage's findTestWorkspaces fails
closed on a workspace that has a test script without a baseline entry, and this
one should be measured. 100% lines, 100% branches. The six existing workspaces
are untouched — they were already reading above their baselines before this
branch, which is pre-existing drift and not for a test refactor to ratchet.

* test(test-support): keep `sql` live, and fail the throwing double at execution

Two defects the first migration batch walked straight into.

`sql` was a getter over `calls`, so `const { db, sql } = fake()` — the natural
way to use it, and what flows.test.ts and authorities.test.ts already wrote
against their hand-rolled spies — captured an empty snapshot that never filled
in. Every later assertion then read nothing and passed for the wrong reason,
which is the exact failure this helper exists to remove. It is a live array kept
in step with `calls` now, and a test pins the destructured form.

throwingD1 threw from prepare(). D1's prepare() is lazy and never touches the
database: a missing table surfaces on all()/first()/run(). A double that failed
earlier would let a test claim it covers an error path it never reaches — and
related-persons.test.ts, whose whole point is that an un-migrated environment
degrades instead of 500ing, hand-rolled a double that threw at execution for
exactly that reason. It now rejects from the three execution methods and records
the statement that failed, so the offending SQL stays inspectable.

* test(db): route the query doubles through the shared fake

Sixteen files in packages/db/src/queries, each of which built its own D1 double
that dispatched on `sql.includes('…')` and fell through to no rows. Fixtures and
assertions are unchanged — only the double moves.

Measured before touching anything, by breaking each marker in the production SQL
and running the test: ELEVEN marker paths across nine files stayed green against
an emptied result. authorities (FROM authority_totals), companies (ORDER BY
bidder_id), competition and trend and flows (FROM sector_totals), contracts
(facet_counts), home (bids_received = 1, JOIN), network (FROM company_totals,
FROM authority_totals WHERE authority_id), search (sqlite_master). Every one of
them now rejects with the marker set it was looking for.

Three things the migration turned up that were not in the issue:

  - regions.test.ts served *region* rows to sectorOptions, which asks a
    completely different table. It reached the same answer only because
    sectorOptions reads r.division, the region fixture has no such field, and
    the filter dropped every row. The route says `all: []` now, and says why.
  - companies.test.ts registered two facet routes for queries no test in it ever
    issues — getCompanyFacets is not exercised there. Dropped rather than kept
    as decoration.
  - companies' CSV stream and list query both read company_totals, so breaking
    the stream's ORDER BY quietly fell through to the list route and returned an
    unpaginated page. They are separated by their own markers now (ORDER BY
    bidder_id vs AS sort_value), and breaking either one throws.

Two markers still survive being broken — authorities' and companies' `FROM
<rollup>`. That is the harness, not the tests: `FROM ${src.from}` is composed at
runtime, so the literal never appears in the source to be mutated. Mutating the
`from:` value itself is caught by both.

Route matching is still substring-based, so a query can fall from a specific
route to a more general one in the same set. What is gone is the *default*
fall-through to emptiness — an unrouted query throws.

packages/db: 487 tests pass; coverage unmoved.

* test(test-support): one real-SQLite D1 facade instead of four copies

d1FromSqlite lived in packages/ingest/src/test/, and packages/db had two
byte-identical re-implementations of it (contracts-filter-sql, value-base-sql,
differing only in a local variable name) while apps/etl reached the original
through ../../../packages/ingest/src/test/d1-sqlite — a relative path across a
workspace boundary, which is what a missing shared home looks like.

It moves to @sigma/test-support beside the fake. The two are different tools and
stay different: this one runs the real SQL against a real node:sqlite database
where SQL semantics are what is under test; fakeD1 is for the TypeScript logic
around a query. Now they at least live in the same place, and the gate's
allowlist names one package instead of two.

Slightly wider than midt-bg#325 asked — the issue scopes itself to the fake doubles and
puts real-SQLite tests out of scope. It is here because "one cast everywhere" is
one of its own done-when boxes, and two of the four remaining casts were these
copies. Reviewer's call; it lifts out cleanly.

Side effect worth noting: d1-sqlite.ts leaves packages/ingest's coverage
denominator by leaving the workspace, which is the outcome midt-bg#254 wanted from a
by-name exclusion, reached by construction instead.

db 487, ingest 84, etl 20 — all pass.

* test(db): recording doubles for the two readonly wrapper suites

readonly-d1 and readonly-corpus test a *wrapper* over D1, not a query: what
matters is which statements reach the handle underneath, not what comes back.
Marker dispatch is the wrong shape for that, so both use recordingD1 — answers
anything, records everything — with `when: []`, a route that constrains nothing.

Two things came out of it, both in the helper:

  - `when: []` matching every query was already true (every() over no markers),
    but undocumented and unpinned. Now both.
  - readonly-d1's hand-rolled log tagged its entries `prepare:` / `exec:`, and
    flattening that into plain SQL would have cost the test its point: a wrapper
    that sent an exec down the prepare path emits identical text, and the
    assertion could no longer tell. FakeD1Call carries `via` now, and the
    corpus's zero-proxy row survives as the response to a constraint-free route.

readonly-corpus also dropped a `raw()` no production path calls.

packages/db: 487 tests pass.

* test(etl): route the ETL doubles through the shared fake

Three doubles. The integrity gate's fake dispatched on eleven markers and fell
through to no rows; it now names all eleven as routes and rejects anything else.
Its local builder was called `fakeD1`, which is the shared helper's name, so it
becomes `servedD1` — which is what it models anyway: a served D1 after
precompute, not any old one.

eop.test.ts also passed `{} as D1Database` twice, for paths that fail before
they reach the database. `fakeD1([])` states that instead of implying it: a
route-less double rejects any query, so if one of those paths ever did reach D1
the test would say so rather than throwing an incidental TypeError on an empty
object.

The freshness double's guard survives as a route that throws its own message —
"raw staging should not be read for planning" is a claim worth keeping in the
test, rather than degrading to the generic no-route error.

apps/etl: 20 tests pass.

* test(test-support): per-route meta, and cover the SQLite facade in its new home

apps/web's assistant tests need meta.rows_read and meta.total_attempts: they
drive the rows-read budget that keeps a retried full scan from under-billing the
Denial-of-Wallet limit (midt-bg#122, review #80). Flattening that to a fixed empty meta
would have quietly removed what those two tests assert, so a route can declare
its own meta. Default stays `{}`.

Moving d1-sqlite.ts here left it with no tests of its own — its callers live in
db, ingest and etl, and none of them count toward this workspace. The ratchet
caught it at 81% and it is covered directly now, including the case nothing
tested anywhere before: batch() rolls back when one statement fails. A
half-applied batch would leave a fixture in a state no production path can
reach, and whoever met it would be debugging a ghost.

While covering it, throwingD1's bind() read `calls.at(-1)` — so binding statement
A after preparing B recorded the arguments against B. Same statement-independence
bug fakeD1 already had a test against; it captures its own record now, and so
does the test.

100% lines, 100% branches, 44 tests.

* test(web): route the last two doubles through the shared fake

assistant/tools built a double whose only real job was carrying meta; it now
declares that meta on a route. csv-export asserted the expected SQL *inside* its
fake — that assertion becomes the route's own marker, so a query that no longer
matches rejects and names both the statement and what was expected, instead of
failing an inline expect from inside a stub.

With these two the gate from the first commit goes green: 279 files scanned, no
D1Database cast outside @sigma/test-support. It opened at 36 casts in 24 files.

apps/web: 493 tests pass.

* fix(test-support): route exec() and batch() through the same contract

batch() recorded each statement and returned a synthetic success without ever
consulting the routes, so a batch of unregistered SQL passed against nothing —
the silent green this helper exists to kill, on the one entry point the write
paths use exclusively (staging, refresh, fx never call prepare().run()).
exec() had the identical hole one method up.

Both now look the statement up. They ask only whether it is registered at all,
not for a particular response shape the way all()/first()/run() do, and throw
naming the SQL and every marker when it is not.

Also: run() and batch() carry the `results` key a real D1Result always has —
the cast to D1Database was hiding its absence; the header no longer points at
the facade's pre-move path; and the second batch() record is documented as a
log of entry points rather than a double count.

* fix(test-support): carry the full D1Result shape in the SQLite facade

all() returned no `meta`, run() neither `meta` nor `results`, batch() no
`results`. The cast to D1Database hid every one of them: the first caller to
read one would get `undefined` from the facade where real D1 hands back `[]`
or `{}`. One D1Shape type spells out all three keys.

* test(web): keep the exact-SQL check csv-export had before the migration

The hand-rolled double asserted `expect(sql).toBe(...)` on the whole statement.
Migrating turned that string into a `when` marker, and markers match by
substring — so the one place the refactor loosened a check rather than
tightening it. Measured: wrapping the production statement leaves all 34 tests
green. The equality moves inside the route, where the callback sees `call.sql`.

* test(ci): pin the gate's scan roots and catch an aliased D1Database

Two ways past the gate, both reproduced. `type DBAlias = D1Database` and then
`as unknown as DBAlias` leaves no D1Database token for the pattern to find; a
renamed type import does the same. A second pass treats giving the type another
name outside the allowlist as the offence, while leaving ordinary annotations
(`db: D1Database`, a field on an Env type) alone.

And SCAN_ROOTS was module-private, so deleting 'apps' from it left the self-test
12/12 green while web and etl dropped out of enforcement. Exported and pinned: a
pattern applied to half the repo is a gate that passes while enforcing nothing.

* fix(test-support): route a batched statement by what it does, not its markers

Markers alone are blind to the method, and it is measurable: a route declaring
only `all:` answered a batched `DELETE FROM staging` with its rows, because
`FROM staging` is a substring of the write. The same SQL through prepare().run()
threw. Narrower than an unrouted batch, but the same silent pass.

A batched write now needs a `run:` route and a batched SELECT an `all:` one;
neither settles for the other, and a SELECT no longer fires a write effect it
happens to match. A write still serves rows when it has them, for RETURNING.
exec() asks for `run:` too — it hands back no rows, so nothing else means
anything to it. That retires `registered()`: every entry point is method-aware.

Reading is decided by the leading keyword, so a `WITH … INSERT` reads as a
SELECT here. That costs a false rejection, never a false pass.

Also: run() carries the route's meta, which batch() already resolved for the
same statement, and batch() documents that it is not transactional — real D1
and the d1-sqlite.ts facade roll back, this does not.

* test(ci): widen the alias rule to where an alias actually goes

The pattern closed three spellings while the comment promised the class. Four
more walked past it: `D1Database & {}`, `Pick<D1Database, …>`, a namespaced
`import('…').D1Database`, and a heritage list naming it off the first position.

The rule is now positional — the mention must sit right of `=`, or inside an
intersection, union, type argument or namespace, never where a parameter or a
field goes. `type Env = { DB: D1Database }` and the conditional type in
readonly-corpus.test.ts stay clean, both pinned.

`implements` is deliberately out: ReadonlyD1 implements D1Database in
production, and TypeScript forces a complete implementation there, so it is no
shortcut to a stub. The comment now says best-effort and means it.

---------

Co-authored-by: Todor Kolev <tkolev@obecto.com>